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"AUGMENTED VIDEO SURVEILLANCE SYSTEM" 



RAKKGROUN " to THE INVENTION 

THIS invention relates to an augmented video surveillance system. 

Recognised problems with known video surveillance systems, whether for 
security or other purposes, are boredom and inefficient or improper reaction 
to specified events by surveillance operators. A further problem is that 
there are cnly limited methods for measuring or monitoring the performance 
of surveillance operators. 

The present invention seeks to address these problems. 



«i IMMARY Q <= ™F INVENTION 

According to the invention there is provided an augmented video 
surveiiiance system in which a surveillance video is -gmented by 
insertion, into Hve.video images seen by an operator of 
realistic objects and/or by visual enhancement of predetermined events 
taking place in the live video image seen by the operator, such insertion 
requiring specific action by the operator, and monitoring the action taken by 
the operator in response to such insertion. 

The objective of the invention is to improve the performance and 
effectiveness of the operator and to measure his performance online In 
accordance with the invention this is achieved by engaging the operator s 
attention by interaction with the live video with the resuit that the operators 
,eve«s of vigilance and interest are elevated. The system proposed by the 
invention may be used to provide on-line operator training. 



The invention also provides apparatus for use in the system summarized 
above. 

BRIEF DES^-R'PTION OF THF DRAWINGS 

The invention will now be described in more detail, by way of example only, 
with reference to the accompanying block diagram which illustrates the 
principles of the invention. 



SPECIFIC DESCRIPTION 

Referring to the diagram, the numeral 10 indicates an event controller 
which controls and implements video augmentation according to the 
invention. The event controller 10 can be programmed to augment a l.ve 
video seen by an operator in a number of different ways which may be 
implemented alone or in combination with one another. 

,n one version live video events are brought to the attention of the operator. 
These are actual live events taking place in the scene observed by the 
operator. Examples of typical events include entry of a person .nto the 
observed scene, addition or removal of inanimate objects to the scene. Th.s 
is indicated by the numeral 1 1 in the diagram. 

As described in a simultaneously filed, co-pending patent application filed 
by the present applicant and entitled "Enhanced vldeo-Based Surveillance 
System" a background image or model Is generated and, during the course 
of live video monitoring. The overall Image is segmented into respects 
background and foreground segments and video enhancement techmques 
are employed visually to enhance all or selected foreground features or 
events, thereby drawing the operator's attention to. such features and 
events. 
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The system described above may be interfaced with a video motion 
detection system also described in the aforementioned co-pending patent 
application, in which case the operator's attention is drawn, by camera 
switching and/or image enhancement to specified motion events In the 
observed image. This is indicated by the numeral 13 in the diagram. 

The operator is trained to respond to specified foreground features or 
events, in each case in a predetermined manner. The system may be used 
to measure the operator's efficiency in responding in the appropriate 
manner to specified features. 

The invention proposes the insertion of certain synthetic or realistic objects 
or events into the image viewed by the operator. In one example the event 
controller 10 acts in response to a rules-based event generation module 12 
to initiate the generation, by a synthetic object generator 14, of a synthetic 
event or object and augments the raw, live video image 16 by inserting 
such event or object into the image, thereby forming an augmented v.deo 
image 18 which is viewed by the operator. In each case the controller 10 
controls all parameters relating to the insertion, eg the frequency of 
insertion, the duration of the insertion and so on. 

Synthetic objects are objects which are inserted into the viewed image and 
are classified as synthetic because they do not blend into the imaged scene 
and can be clearly identified by the operator as artificially inserted objects. 
Typical examples are fanciful shapes such as stars, squares, circles or still 
pictures of persons. In each case the insertion may require specific acton 
by the operator, as indicated by the numeral 20. 

Where the live video seen by the operator is augmented by insertion of an 
inanimate, fanciful image such as a star, circle or square, the operator may 
be required to operate a mouse pointer and to click on the insertion in order 
to remove it. The operator's reaction time and ability to carry out the 
required action can be monitored to provide an indication of the operator's 
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efficiency and training and can, if necessary, be logged as part of the 
operator's performance record. 

Different categories of synthetic objects, each requiring a different reaction 
by the operator, can be inserted. For instance, still pictures or Images of 
persons may show such persons displaying normal normal, unsuspicious 
behaviour and may require a first type of response by the operator. On the 
other hand, pictures or images of persons displaying suspicious behaviour 
might require a totally different type of response by the operator. The 
augmentation of the live video by the insertion of different types of synthetic 
image may thus require the operator to have the ability to distinguish 
between different events and to act accordingly in response to the specific 
event with which he is confronted. Once again, the operator's reaction to 
the inserted object(s) can be used to provide a measure of his ability and 
efficiency. 

The object insertion technique described in the preceding paragraph also 
serves a valuable function in on-line training of the operator in that it can 
raise the operator's awareness of certain types of personal behaviour which 
may be considered undesirable from a security point of view, for example a 
person assuming a certain bodily posture in the vicinity of a high-risk area 
such as a cash-counting or gem sorting station. 

The invention also envisages the augmentation of the live video by 
insertion of realistic objects or events, under the control of the event 
controller 10 and in accordance with the rules-based event generat.cn 
module 12. In this case, realistic video objects, such as an.mated 
sequences extracted from previously recorded images of the scene under 
surveillance, as indicated by the block 22 in the diagram, are inserted back 
into the live video image 16 to produce the augmented image 18 in such a 
manner that the operator is unable to distinguish between artificially 
inserted objects and real objects of the scene. So, for instance, live v.deo 
footage of a person moving in the surveilled scene can be Inserted to 
produce the augmented image. 
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The operator will once again be required to react in a certain way to the 
augmented video which he sees. Where, for instance, augmentataon 
involves insertion of video footage showing an act of theft, the techn.que 
can be used to monitor the operator's ability to recognize the act and to 
react accordingly. 

In each of the examples described above, the operator will be required to 
interact, in accordance with his training, via an interface generator 24. 
Where a synthetic object has been inserted, the operator may for example 
be required to click once, with a mouse pointer, on an image such as a 
fanciful image or a realistic image of non-threatening type or to dick twice 
on an image representing a security threat. The event controller may be 
programmed to give feedback to the operator, via the interface 24, to 
indicate that the correct action has been taken. The inserted object may for 
instance be cause to flash and/or to disappear from the image, thereby 
acknowledging the correctness of the operator's action. 
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